(1) Field of the Invention
The present invention relates to an image decoding and encoding apparatus and an image encoding method which is suitable for transmission of compressed motion image data and for reproduction of expanded image data in the receiving side.
(2) Description of the Related Art
In a so-called image signal transmission system for transmitting a motion image signal to distant areas such as a TV conference system or a TV telephone system, an image signal has been encoded utilizing line correlation or frame correlation of the image signal and thereby transmission efficiency of significant information has been enhanced in order to efficiently utilize the transmission path.
FIGS. 1, 1(A), 1(B) and 1(C) are block diagrams illustrating an example of structure of an image encoding apparatus of the related art for encoding image data. An image data to be encoded is inputted to a motion vector detecting circuit 1. The motion vector detecting circuit 1 processes image data of each frame as the I picture, P picture or B picture in accordance with the preset sequence. It is predetermined that as which picture of the I, P, B pictures an image of each frame sequentially inputted should be processed. An image data of the frame to be processed as the I picture is stored in a forward original image part 2a, while an image data processed as the B picture is stored in a reference original image part 2b and an image data to be processed as the P picture is stored in a backward original image part 2c.
Moreover, when an image of the frame to be processed as the B picture or P picture, the image data of the first P picture having been stored in the backward original image part 2c is transferred to the forward original image part 2a, the image data of the next B picture is stored in the reference original image part 2b and the image data of the next P picture is stored in the backward original image part 2c (overwriting). Such operations are repeated.
The motion vector detecting circuit 1 divides an image data of the frame of the B picture stored in the reference original image part 2b in unit of block of 8.times.8 pixels and detects motion vectors between the I picture image stored in the forward original image part 2a and the P picture image stored in the backward original image part 2c. The motion vector is not detected for the I picture.
The motion vector detecting circuit 1 divides an image data of the frame of the P picture stored in the reference original image part 2c in units of block of 8.times.8 pixels and detects motion vectors between the I picture image stored in the forward original image part 2a and the P picture image stored in the backward original image part 2c. The motion vector is not detected for the I picture.
The motion vector detecting circuit 1 outputs the image data of which motion vector is detected in units of blocks to an arithmetic operation part 3 in the next stage in units of macroblock.
That is, each frame image data is divided into N slices as shown in FIG. 2(a) and each slice is determined to include M macroblocks as shown in FIG. 2(b). Each macroblock is formed by luminance signal data Y1 to Y4 and corresponding color difference signal data Cb and Cr of the block in units of 8.times.8 pixels as shown in FIG. 2(c).
In this case, image data in the slice is arranged continuously in units of macroblocks so that image data are appear continuously in units of blocks within this macroblock in the sequence of raster scanning.
Here, the macroblock uses, as the one unit, image data (Y1 to Y4) of the 16.times.16 pixels continuously arranged in the horizontal and vertical scanning directions for the luminance signal, while the amount of data is reduced for corresponding two color difference signals Cb, Cr and the one block of 8.times.8 pixels is respectively assigned to such color difference signals.
The motion vector detecting circuit 1 also outputs the motion vectors of four blocks of each macroblock to a variable length encoding circuit 6 and a motion compensating circuit 13 and also obtains a sum of the absolute values of such motion vectors and then outputs a result to a prediction deciding circuit 14.
The prediction deciding circuit 14 sets an intra-frame prediction mode (mode without motion compensation) as a prediction mode when the motion vector detecting circuit 1 is reading image data of the I picture from the forward original image part 2a (when a sum of absolute values of motion vectors supplied from the motion vector detecting circuit 1 is 0) and changes over a switch 3d of the arithmetic operation part 3 to the contact a. Thereby, an image data of the I picture is inputted to a DCT circuit 4 and is transformed to a DCT coefficient through the DCT (Discrete Cosine Transformation) processing. This DCT coefficient is then inputted to a quantizing circuit 5 and thereafter inputted to a variable length encoding circuit 6 after quantization in the quantizing step corresponding to the amount of data accumulated in the send buffer 7 (buffer accumulation amount).
The variable length encoding circuit 6 converts an image data (in this case, the data of I picture) supplied from the quantizing circuit 5 into a variable length code, for example, Huffman code corresponding to a quantizing step supplied from the quantizing circuit 5, a prediction mode supplied from the prediction deciding circuit 14 and a motion vector supplied from the motion vector detecting circuit 1 and then outputs such variable length code to the send buffer 7.
The send buffer 7 temporarily accumulates the input data and then outputs it to a transmission data control circuit 63. A transmission data control circuit 111 outputs the data supplied from the send buffer 7 to the transmission path.
Meanwhile, data of I picture outputted from the quantizing circuit 5 is then inputted to an inverse quantizing circuit 9 and is inversely quantized corresponding to the quantizing step supplied from the quantizing circuit 5. An output of the inverse quantizing circuit 9 is inputted to an IDCT (inverse DCT) circuit 10 for the inverse DCT processing. Thereafter, the output signal is supplied to and stored in a forward prediction image part 12a of a frame memory 12 through the arithmetic operation part 11.
In the case of processing image data of frames frame to be inputted sequentially, for example, as the I, B, P, B, P, B, . . . pictures, the motion vector detecting circuit 1 processes the image data of the frame inputted first as the I picture and thereafter processes the image data of the second next input frame as the P picture before processing the image data of the next input frame as the B picture. It is because the B picture must be followed by the backward prediction and cannot be decoded if P picture is not prepared precedingly as the backward prediction image.
Therefore, the motion vector detecting circuit 1 detects, after the I picture, the motion vector of the image data of the P picture stored in the backward original image part 2c in units of the block of 8.times.8 pixels. Then, a sum of the absolute values of motion vectors of the block consisting of four 8.times.8 pixels forming a macroblock is supplied to the prediction deciding circuit 14 from the motion vector detecting circuit 1. The prediction deciding circuit 14 sets, as a prediction mode, the intra-frame prediction mode when the sum of absolute values of the macroblock of this P picture is smaller than the preset reference value. Moreover, when the sum is larger than the reference value, the prediction deciding circuit 14 sets the forward prediction mode.
The arithmetic operation part 3 changes over the switch 3d to the contact a as explained above when the intra-frame prediction mode is set. Therefore, this data is transmitted to the transmission path, like the data of the I picture, through the DCT circuit 4, quantizing circuit 5, variable length encoding circuit 6, send buffer 7 and transmission data control circuit 111. Moreover, this data is supplied to and stored in the backward prediction image part 12b of the frame memory 12 through the inverse quantizing circuit 9, IDCT circuit 10 and arithmetic operation part 11.
In the forward prediction mode, the switch 3d is changed over to the contact b. Thereby, an image (the image of the I picture, in this case) data stored in the forward prediction image part 12a is read and is then subjected to motion compensation, by the motion compensating circuit 13, corresponding to the motion vector outputted from the motion vector detecting circuit 1. That is, the motion compensating circuit 13 reads, when setting of the forward prediction mode is instructed from the prediction deciding circuit 14, the data to generate the prediction image data by shifting the read address of the forward prediction image part 12a as much as the amount corresponding to the motion vector from the position corresponding to the position of macroblock which is now outputted from the motion vector detecting circuit 1.
The prediction image data outputted from the motion compensating circuit 13 is then supplied to a arithmetic operation part 3a. The arithmetic operation part 3a subtracts the prediction image data, supplied from the motion compensating circuit 13, corresponding to the macroblock supplied from the motion vector detecting circuit 1 from the data of this macroblock in order to output a difference between these data. This difference data is transmitted to the transmission path through the DCT circuit 4, quantizing circuit 5, variable length encoding circuit 6, send buffer 7 and transmission data control circuit 111. Moreover, this difference data is inputted to the arithmetic operation part 11 through the inverse quantizing circuit 9 and IDCT circuit 10.
To this arithmetic operation part 11, the data identical to the prediction image data supplied to the arithmetic operation part 3a is also supplied. The arithmetic operation part 11 adds the prediction image data outputted from the motion compensating circuit 13 to the difference data outputted from the IDCT circuit 10. Thereby, the image data of original P picture can be obtained. The image data of this P picture is supplied to and stored in the backward prediction image part 12b of the frame memory 12.
As explained above, after the data of the I picture and P picture are respectively stored in the forward prediction image part 12a and backward prediction image part 12b, the motion vector detecting circuit 1 detects the motion vector of B picture in units of blocks. The prediction deciding circuit 14 sets the prediction mode to any one of the intra-frame prediction mode, forward prediction mode, backward prediction mode or both-direction prediction mode in accordance with a sum of absolute values of motion vectors of blocks forming the macroblock.
As explained previously, the switch 3d is changed over to the contact a or b during the intra-frame prediction mode or forward prediction mode. In this case, the processing similar to that for the P picture is also executed for transmission of data.
On the other hand, when the backward prediction mode or both-direction prediction mode is set, the switch 3d is changed over to the contact c or d, respectively.
During the backward prediction mode where the switch 3d is changed over to the contact c, an image (image of P picture in this case) data stored in the backward prediction image part 12b is read and motion is compensated by the motion compensating circuit 13 corresponding to the motion vector outputted from the motion vector detecting circuit 1. That is, the motion compensating circuit 13 reads, when setting of the backward prediction mode is instructed from the prediction deciding circuit 14, the data to generate the prediction image data by shifting the read address of the backward prediction image part 12b as much as the amount corresponding to the motion vector from the position corresponding to the position of macroblock which is now outputted from the motion vector detecting circuit 1.
The prediction image data outputted from the motion compensating circuit 13 is supplied to an arithmetic operation part 3b. The arithmetic operation part 3b subtracts the prediction image data supplied from the motion compensating circuit 13 from the data of macroblock supplied from the motion vector detecting circuit 1 in order to output a difference between these data. This difference data is transmitted to the transmission path through the DCT circuit 4, quantizing circuit 5, variable length encoding circuit 6, send buffer 7 and transmission data control circuit 111.
During the both-direction prediction mode where the switch 3d is changed over to the contact d, an image (image of the I picture in this case) data stored in the forward prediction image part 12a and an image (image of the P picture in this case) data stored in the backward prediction image part 12b are read and are subjected to motion compensation by the motion compensating circuit 13 corresponding to the motion vector outputted from the motion vector detecting circuit 1. Namely, when setting of both-direction prediction mode is instructed from the prediction deciding circuit 14, the motion compensating circuit 13 reads data to generate the prediction image data by shifting the read address of the forward prediction image part 12a and backward prediction image part 12b as much as the amount corresponding to the motion vector from the position corresponding to the position of macroblock which is now outputted from the motion vector detecting circuit 1.
The prediction image data outputted from the motion compensating circuit 13 is supplied to an arithmetic operation part 3c. The arithmetic operation part 3c subtracts the prediction image data supplied from the motion compensating circuit 13 from the data of macroblock supplied from the motion vector detecting circuit 1 to output a difference. This difference data is transmitted to the transmission path through the DCT circuit 4, quantizing circuit 5, variable length encoding circuit 6, send buffer 7 and transmission data control circuit 111.
An image of B picture is never used as the prediction image of the other image and therefore it is not stored in the frame memory 12.
As explained above, since the image data is transmitted as the variable length code, when a simple stationary image, for example, continues for a comparatively longer period, the data to be transmitted sometimes becomes shortage. In this case, in view of preventing missing of transmission data, an invalid code can be added to the data to be transmitted. This invalid code can also be added, for example, in units of slice or macroblock shown in FIGS. 2(a) to 2(c).
FIG. 3 illustrates an example where an invalid code (invalid data) is added in units of slice. Each slice is provided with a slice start code at its leading area. This slice start code is formed by a synchronous code and an attribute code. The synchronous code is formed by the data of two bytes where each bit is all set to logic 0 and the data of one byte (three bytes in total) where LSB is set to logic 1 and the other bits are set to logic 0. Moreover, the attribute code is set to one byte where the code indicating the data concerning the slice data such as the attribute of corresponding slice is arranged. Therefore, the slice start code is formed by the data of 4 bytes in total (32 bits).
This slice start code is added in a unit of slice without relation to shortage of the data to be transmitted.
The invalid code is added in such a manner that the data where all bits are set to logic 0 is added before the slice start code as many as required in unit of the byte (8 bits). This code is added in units of slice only when there is a shortage of the data to be transmitted.
FIG. 4 illustrates an invalid code to be added to c block. Namely, in this case, total of 11 bits where upper 7 bits are set to logic 0 and lower 4 bits to logic 1 are considered as a unit of the invalid code and this invalid code is added before a valid code of the macroblock as many as the predetermined number of units.
In the case where an invalid code is added to the data of macroblock as shown in FIG. 4, the transmission data control circuit 111 shown in FIG. 1 can be structured, for example, as shown in FIG. 5. In this example, the data outputted from the send buffer 7 is inputted to an N/M converter 121 and thereby the data divided in units of N bits is converted to the data which is divided in units of M bits. The data outputted from the N/M converter 121 is inputted to a multiplexer (MUX) 122 and is combined therein with the invalid code outputted by th e macroblock invalid code generating circuit 123.
That is, the macroblock invalid code generating circuit 123 generates an invalid code shown in FIG. 4 where the upper 7 bits are set to logic 0, while the lower 4 bits to logic 1 and outputs this invalid code to the multiplexer 122. A controller 124 controls the multiplexer 122 corresponding to the send buffer information and selects, when the send buffer 7 does not generate underflow condition, an output of the N/M converter 121 or selects an invalid code outputted from the macroblock invalid code generating circuit 123 when the send buffer 7 is supposed to generate underflow condition. Therefore, the data outputted from the multiplexer 122 mixes invalid codes of the desired number of units.
Next, FIGS. 6, 6(A) and 6(B) are block diagrams illustrating an example of structure of an image decoding apparatus for decoding data encoded by the image encoding apparatus of FIG. 1. The encoded image data transmitted through the transmission path is received by a receiving circuit not illustrated, temporarily stored in a receiving buffer 32 and is then supplied to a variable length decoding circuit 33 of a decoding circuit 50. The variable length decoding circuit 33 decodes the data supplied from the receiving buffer 32 by the variable length decoding method and respectively outputs the motion vector and prediction mode to the motion compensating circuit 38 and the quantizing step to the inverse quantizing circuit 34 and also outputs the decoded (by variable length decoding) image data to the inverse quantizing circuit 34.
The inverse quantizing circuit 34 inversely quantizes the image data supplied from the variable length decoding circuit 33 in accordance with the quantizing step which is also supplied from the variable length decoding circuit 33 and outputs the quantized data to the IDCT circuit 35. The data (DCT coefficient) outputted from the inverse quantizing circuit 34 is subjected to the inverse DCT processing in the IDCT circuit 35 and is then supplied to the arithmetic operation part 36.
In the case where the image data supplied from the IDCT circuit 35 is the data of I picture, such data is outputted from the arithmetic operation part 36 and it is supplied to and stored in the forward prediction image part 37a of the frame memory 37 in order to generate a prediction image data of the image data (data of P or B picture) inputted rater to the arithmetic operation part 36.
Moreover, this data is supplied to a display 40 for displaying purpose after it is subjected to the D/A conversion in a D/A converter 39.
In the case where the image data supplied from the IDCT circuit 35 is the data of P picture having the image data of the preceding frame as the prediction image data, the image data (data of the I picture) of the preceding frame stored in the forward prediction image part 37a of the frame memory 37 is read and is subjected to motion compensation, in the motion compensating circuit 38, corresponding to the motion vector outputted from the variable length decoding circuit 33. In the arithmetic operation part 36, such image data is added to the image data (difference data) supplied from the IDCT circuit 35 before the added data is outputted. The added data, namely the decoded data of P picture is supplied to and stored in the backward prediction image part 37b of the frame memory 47 in order to generate the prediction image data of the image data (data of B picture) to be inputted rater to the arithmetic operation part 36.
Since this P picture is the image to be displayed after the next B picture, it is not yet displayed in this point.
In the case where the image data supplied from the IDCT circuit 35 is the data of B picture, the image data (forward prediction mode) of the I picture stored in the forward prediction image part 37a of the frame memory 37, the image data (backward prediction mode) of the P picture stored in the backward prediction image part 37b or both image data (both-direction prediction mode) are read corresponding to the prediction mode supplied from the variable length decoding circuit 33 and are subjected to the motion compensation in the motion compensating circuit 38 corresponding to motion vector outputted from the variable length decoding circuit 33.
As explained above, data after the motion compensator in the motion compensating circuit 38 is added with an output of the IDCT circuit 35 in the arithmetic operation part 36. This added output is supplied on the display 40 for the displaying purpose after D/A conversion in the D/A converter 39.
However, since this added output is the data of B picture, it is not used for generation of the prediction image of the other images and therefore not stored in the frame memory 37.
After the image of B picture is outputted and displayed, the image data of P picture stored in the backward prediction image part 37b is read and is then supplied to the arithmetic operation part 36 through the motion compensating circuit 38. But, in this case, motion compensation is not carried out. This data is outputted to the display 40 through the D/A converter 39 and displayed.
When an invalid code is added, this invalid code is eliminated in the variable length encoding circuit 33.
In the conventional apparatus, the added invalid code is removed as explained above as a part of the data decoding process (variable length decoding) in the variable length decoding circuit 33. As a result, while the invalid code is being removed in the variable length decoding circuit 33, data is not supplied to each circuit after the inverse quantizing circuit 34 in the successive stages, bringing about a disadvantage that these circuits are not used for the processing. In the case of the NTSC system, an image of one frame is displayed on the displays 40 in the period of 1/30 second. But, if the invalid code is long, each circuit after the inverse quantizing circuit 34 cannot process the data of one frame within the period of 1/30 second and display of image on the display 40 has been discontinued.